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(57) An apparatus and method for correctly deter- 
mining the position of a vehicle in a traffic lane by ob- 
taining correct infomiatlon about the position of the traf- 
fic lane without being affected by variations In the road 
surface, weather, time of day, or such imaging condi- 
tions as fixed or moving lighting, are provided. An edge 
signal (SH) of a high spatial frequency component and 
a luminance signal (SL) of a low spatial frequency com- 
ponent of a digital image signal (Si) representing the 
view of the local area (Vi) to the front of a vehicle are 
extracted. A road contour signal (Sre) is then extracted 
from the edge signal (SH), and a road region signal (Sn-) 
is extracted from the luminance signal (SL). The position 
of the lane Sre Is then detected with high precision by 
evaluating the lane contour Sre based on the road re- 
gion signal Sn' and lane contour data Sre. 
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Description 

BACKGROUND OF THE INVENTION 

5 1 . Field of the Invention 

[0001] The present invention relates to a local positioning apparatus used in a local positioning system particularly 
suited to detecting the condition of a subject, such as an automobile or other motor vehicle, on a road while either 

stationary or moving, based on such local positioning information as the relative location, velocity, and attitude of the 
10 subject within a localized region with reference to an image obtained by imaging the area in front of said subject. IVlore 
specifically, the present invention relates to a local positioning apparatus for correctly detecting the position of a subject 
within a localized region without being affected by changes in the condition of the road surface, weather, time of day, 
fixed lighting, moving lighting, and other changes in the imaging condition. 

[0002] A local positioning apparatus according to the priorart used in an automobile is shown in Fig. 25 and described 
'5 below. As shown in Fig. 25, this conventional local positioning apparatus LPP comprises an edge extractor 1 P, threshold 
generator3P,contourextractor5P, matching operatorQP, lane (mari<er) contour extractor11P, region llmiter13P, current 
position detector 15P, curvature detector 17P, and yaw angle detector 19R 

[0003] The edge extractor 1 P is connected to a digital imaging apparatus 100 (Rg. 1). The digital imaging apparatus 

1 00 is mounted to the subject, which in this explanation of the prior art and the specification of the present invention 
20 below is, by way of example only an automobile AlVi (Fig. 4), for capturing a perspective image Vi of the view to the 
front of the automobile AM in the direction of travel, and generating a digital image signal Si of the perspective image 
vi. included in the perspective image Vi are an image of the road, the lane markers Lml and Lm2 defining the boundaries 
(sides) of the lane Lm in which the automobile AM Is currently travelling, and a lane marker Lm3 defining the far 
boundary of an adjacent lane (Fig. 5). The edge extractor 1 P extracts the edge pixels of the lane markers Lm1 , Lm2, 
2s and Lm3 from the digital image signal Si, and generates an extracted edge pixel signal Sx". The extracted edge pixel 
signal Sx' contains only the edge pixels extracted by the edge extractor 1 P, and thus represents an extracted edge 
image Vx'. 

[0004] Using a known method, the threshold generator 3P scans the extracted edge pixel signal Sx' to extract a line 
for each of the lanes Lm delineated by the lane markers Lml, Lm2, and Lm3, and determines a threshold value Eth' 
30 lor extracting pixels representing the contours of the lane markings from the extracted edge pixel signal Sx'. Using the 
supplied threshold value Eth', the contour extractor 5P then extracts contour pixels and generates an extracted contour 
signal So' representing the contour lines of the lane markings. 

[0005] The matching operator 9P then detennines the line segment or arc matching the contour lines contained in 
the extracted contour signal So', and generates matching data Sm' containing ail of the matching line and arc segments. 
35 [0006] The iane contour extractor 1 1 P then compares the matching data Sm' with typical lane dimension character- 
istics stored in memory to extract the matching line elements meeting these dimensional criteria as the contour lines 
of the lane, and outputs the result as lane extraction signal Smc'. 

[0007] Based on this lane extraction signal Smc', the region limiter 1 3P defines a certain region around the extracted 
iane, and generates a region signal Sr' delimiting this lane region. By feeding this region signal Sr' back to the edge 
40 extractor 1 P, the edge extractor 1 P limits the area within the perspective image VI used for edge extraction to the region 
limits defined by the region limiter 13P. 

[0008] Using the iane extraction signal Smc' from the lane contour extractor 11 P, the current position detector 15P 

detects the position of the automobile AlVl on the road, or more specifically in relationship to the lane being followed. 
[0009] The curvature detector 1 7P detects the curvature of the lane being followed while the automobile AM is mov- 
45 ing. The yaw angle detector 1 9P detects the angle of the automobile AM relative to the lane, i.e. , whetherthe automobile 
AM is travelling parallel to the sides of the lane or is following a course while would result in the automobile AM leaving 
the cun-ent iane being followed. 

[0010] It should be noted that ail of the processes described above are based on the perspective image Vi of the 
area to the front of the vehicle obtained by the digital imaging apparatus 100. It is obvious, however, that correct 

so information about the lane dimensions cannot be obtained from the perspective image Vi because the perspective 
image is a simple two-dimensional representation of three-dimensional space. Specifically, shapes change in the per- 
spective image Vi as the distance from the digital imaging apparatus 100 Increases with the size of an object at a 
distance from the digital imaging apparatus 1 00 being displayed smaller than the same object in close proximity to the 
digital imaging apparatus 100. In addition, the edges of a road or lane are Indistinct in a perspective image Vi, making 

55 edge detection difficult. 

[0011] Road conditions are also not constant, and this further complicates road edge detection. For example, rec- 
ognizing the contour of a road or lane by means of edge detection is, in fact, impossible when the side of a road or a 
lane marker is hidden by such as vegetation, dirt, or gravel. Edge detection is also not practically possible when the 
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lane markers are not recognizable in part or in full because of soiling, damage, or other cause. 
[0012] The brightness of the road surface is also extremely variable, and is affected by such factore as the weather, 
whether the road is inside a tunnel, and whether it is day or night. Stationary lights installed in tunnels and beside the 
road also only partially and locally illuminate the road surface, and spots of extreme brightness or darkness can result 

5 even from the same road surface. These conditions are further complicated at night by irregular changes in illumination 
resulting both from the headlights of the subject automobile AM and the headlights of other vehicles. Accurately de- 
tennining the edge detection threshold value Eth' is effectively Impossible under environments subject to changes In 
driving conditions, time of day, and weather, as well as dynamic changes in brightness in the perspective Image VI due 
to fixed or moving lighting even when the driving conditions, time, and weather remain constant. 

10 [0013] in other words, it is not possible to obtain accurate dimensional infonnation about the road and lane from an 
extracted contour signal Sc' that is based on such an inaccurate threshold value Eth'. 

[0014] It Is therefore clearly extremely dangerous to detect the local positioning of a vehicle relative to a road or lane 
based on such unreliable, inaccurate, and distorted dimensional information, and to detect the road curvature and 
vehicular yaw based on such erroneously detemiined positioning Information. 

IS 

SUMMARY OF THE INVENTION 

[001 5] The object of the present Invention Is therefore to resolve the above problems by providing a local positioning 
apparatus for detecting the local position of a subject capable of advancing in a direction related to a lane within a 

20 localized region based on a digital image signal representing a localized region in the direction of subject travel. To 
achieve said object, said local positioning apparatus comprises a first image signal generating means for extracting a 
high frequency component of a spatial frequency from said digital Image signal to generate an edge signal; a contour 
extracting means for extracting lane contours based on the edge signal, and generating lane contour data; a second 
image signal generating means for extracting a low frequency component of a spatial frequency from said digital image 

25 signal to generate a luminance signal; a lane area extracting means for extracting the lane area based on the luminance 
signal, and generating lane area data; and a lane detecting means for detecting a lane position and generating a lane 
detection signal based on the lane contour data and lane area data. 

BRIEF DESCRIPTION OF THE DRAWINGS 

30 

[0016] These and other objects and features of the present invention will become clearfrom the following description 
taken in conjunction with the preferred embodiments thereof with reference to the accompanying drawings throughout 
which like parts are designated by like reference numerals, and in which: 

35 Fig. 1 is a block diagram of a local positioning apparatus according to a first embodiment of the present invention. 

Fig. 2 is a detailed block diagram of essential components of a local positioning apparatus shown in Fig. 1 , 
Fig. 3 is used to describe the coordinate system of a perspective image in the present Invention, 
Fig. 4. is used to describe the coordinate system of a birds'-eye Image in the present invention. 
Fig. 5 is a perspective image of the view to the front of the automobile AM. that is processed by the local positioning 

40 apparatus of the Invention, 

Fig. 6 shows the edges extracted from the perspective image in Fig. 5, 

Rg. 7 is used to described a method of extracting contours from the edge image in Fig. 5, 

Fig. 8 shows the contour lines extracted from the edge image in Fig. 5, 

Fig. 9 is used to describe how the processing region of the perspective image is limited by the contour extraction 
4S region iimiter in Fig. 2, 

Fig. 1 0 is used to describe a perspective image after region limited by the method shown in 9, 

Fig. 11 Is used to describe the luminance image extracted from the front perspective image shown In Rg. 5, 

Fig. 12 is a graph of the relationship between pixel luminance in the luminance image in Fig. 11 and the pixel count 

having a particular luminance value, 
50 Fig. 13 is a graph of experimental values showing the relationship between the luminance values of pixels in the 

road area of the luminance image, and the focal distance of the road surface from which the corresponding pixel 

was captured, 

Fig. 1 4 is used to describe the road brightness setting method of the road brightness setter shown in Fig. 2, 
Fig. 1 5 is used to describe the road area extraction method of the road Image extractor in Rg. 2, 
55 Fig. 16 is used to describe a method of extracting the road area edges from the road area pixels extracted as 

shown in Fig. 15, 

Fig. 17 shows the road image atter linear approximation using the method shown in Fig. 16, 
Fig. 18 Is used to describe the operation of the road region setter. 
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Fig. 1 9 shows a birds'-eye image after coordinate conversion of the road contour signai by the coordinate converter 
in Fig. 2, 

Fig. 20 is used to describe the curve patterns obtained by Hough conversion of the birds'-eye contour signal Sec 

in Fig. 19, 

5 Fig. 21 is an alternative exampie of the curve patterns obtained by Hough conversion of the birds'-eye contour 

signal Sec in Fig. 19, 

Fig. 22 is an alternative example of the curve patterns obtained by Hough conversion of the birds'-eye contour 
signal Sec in Fig. 19, 

Fig. 23 Is used to describe In-lane position detection by the In-lane position detector, 
10 Fig. 24 Is a flow chart used to describe the overall operation of the local positioning apparatus shown in Fig. 1 , 

Fig. 25 is a biocl< diagram of a local positioning apparatus according to the prior art, 

Fig. 26 Is a block diagram of a road area extraction apparatus according to a second embodiment of the present 
invention, 

Fig. 27 Is an example of a brightness histogram generated from the road image by the brightness histogram gen- 
's erator in Fig. 26, 

Fig. 28 Is a flow chart used to describe the overall operation of the road area extraction apparatus shown in Fig. 26, 
Fig. 29 Is used to describe the method of detection lane edges from the extracted lane area by means of the road 
area extraction apparatus shown In Fig. 26, 

Fig. 30 Is a block diagram of a road area extraction apparatus according to a first alternative of second embodiment 
20 of the present invention, 

Fig. 31 is used to describe the area occupied by the lane area in the brightness histogram generated by the road 
area extraction apparatus in Fig. 30, 

Fig. 32 Is used to describe the brightness distribution obtained by segmenting the road image by means of the 
lane image separator, and the brightness distribution pattern obtained by the brightness distribution pattern detector 

25 in Fig. 30, 

Fig. 33 is a flow chart used to describe the overall operation of the road area extraction apparatus shown in Fig. 30, 
Fig. 34 is a block diagram of a road area extraction apparatus according to a second alternative of the second 
embodiment of the present Invention, 

Fig. 35 Is an example of a brightness histogram generated from the road image by the brightness histogram gen- 
30 erator in Fig. 34, 

Fig. 36 is a flow chart used to describe the overall operation of the road area extraction apparatus shown in Rg. 34, 
Fig. 37 Is a block diagram of a road area extraction apparatus according to a third alternative of the second em- 
bodiment of the present Invention, 

Fig. 38 is an exampie of a brightness histogram generated from the road image by the brightness histogram gen- 
35 erator in Fig. 37, 

Fig. 39 Is used to describe road Image segmentation by the Image separator 43 shown in Fig. 37, 

Fig. 40 Is the first part of a flow chart used to describe the overall operation of the road area extraction apparatus 

shown in Fig. 37, 

Fig. 41 is the second part of a flow chart used to describe the overall operation of the road area extraction apparatus 
40 shown in Fig. 37, 

Fig. 42 is a block diagram of a road area extraction apparatus according to a fourth alternative of the second 
embodiment of the present invention. 

Fig. 43 is used to describe the relationship between extracted lane area obtained by the road area extraction 
apparatus shown in Fig. 42, and a triangle fomied in the image from the vanishing point and road edge pixels, 
45 Fig. 44 is a flow chart used to describe the overall operation of the road area extraction apparatus shown in Fig. 

42, and 

Fig. 45 is a block diagram of a road area extraction apparatus according to the prior art. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

50 

[0017] Fig. 1 is a block diagram of a local positioning apparatus LP according to a first embodiment of the present 
Invention. This local positioning apparatus LP Is used on board motor vehicles such as automobiles AM and other 
types of vehicles that are capable of traversing a freely defined path of travel. 

[0018] As shown in Fig. 1 , the local positioning apparatus LP comprises a digital Imaging apparatus 100, spatial 

55 frequency separator 200, lane area detector 300, lane contour detector 400, lane detector 500, and electronic control 
unit (ECU) 700. Note that the ECU 700 Is a device commonly used and known In the automobile industry, and Is used 
to detect the vehicle condition as represented by th e speed of travel and steering condition, generate a vehicle condition 
signai Sc, which includes a velocity signal Sv and steering signal Ss, and controls the various electrical devices of the 
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vehicle. 

[001 9] The digital imaging apparatus 1 GO comprises a Ph x PV pixel matrix of imaging elements, and continuously 
captures images of the subject, i.e., the area to the front in the direction of travel, where Ph is the number of horizontal 
pixels and PV is the number of vertical pixels. Increasing pixel counts Ph and PV can be used to improve image 
5 resolution, and to therefore improve the positioning precision of the present invention. Increasing the pixel count, how- 
ever, also Increases the manufacturing cost, and the pixel count must therefore be determined by balancing the required 
resolution against the manufacturing cost. While a digital imaging apparatus 100 according to an exemplary embodi- 
ment of the invention described below has a 428 (Ph) by 268 (PV) pixel imaging device, the invention shall obviously 
not be so limited. 

10 [00201 The digital imaging apparatus 1 00 is mounted toward the front of the automobile AM, and from this position 
continuously Images a view Vi In-the direction of travel of the automobile AM to generate a digital image signal Si. This 
view Vi is a perspective image Vi of the area to the front of the automobile AM as seen from the automobile AM. 
Furthermore, this perspective image Vi can be either a still image or a motion picture. 

[0021 ] The spatial frequency separator 200 is connected to the digital imaging apparatus 1 00, and receives therefrom 
15 the digital image signal Si. The spatial frequency separator 200 thus extracts a low frequency component from the 
digital image signal Si to generate a low spatial frequency signal SL, and extracts a high frequency component to 
generate a high spatial frequency signal SH. 

[0022] The lane area detector 300 Is connected to the spatial frequency separator 200, and receives therefrom the 
low spatial frequency signal SL, based on which the lane area detector 300 detects the road surface from the perspec- 

20 five image Vi to generate a road region signal Srr. 

[0023] The lanecontour detector 400 is likewiseconnected to the spatialfrequencyseparatDr200, receives therefrom 
the high spatial frequency signal SH, thus detects the contour of the. road from the perspective image VI, and generates 
a road contour signal Sre. The lane contour detector 400 also generates a region limiting signal Sr that limits the 
scanning area usedforcontourdetection, and inputs this signal to the spatialfrequencyseparator200. More specifically, 

25 the region limiting signal Sr Is used to limit the area In the perspective Image Vi that is scanned for contour detection 
to the area around the detected contour as a means of reducing the processing load. The spatial frequency separator 
200 then extracts the high spatial frequency signal SH only from the area of the digital Image signal Si specified by the 
region limiting signal Sr. 

[0024] The lane detector 500 is connected to the lane area detector 300 and the lane contour detector 400, and thus 
30 receives both the road region signal Srr and the road contour signal Sre. Based on the road region signal Srr, the lane 

detector 500 determines whether the road contour indicated by the road contour signal Sre is con-ect, and generates 
a road judgement signal Sj indicating the result of this evaluation. The road judgement signal Sj Is then fed back to the 
lane contour detector 400. If and only if the lane detector 500 detennlnes that the road contour Indicated by the road 
contour signal Sre is correct does the lane detector 500 pass the road contour signal Sre to the local position detector 

35 600. 

[0025] The lane contour detector 400 updates the road contour signal Sre based on the feedback of the road judge- 
ment signal Sj, and again outputs the road contour signal Sre to the lane detector 500. The lane contour detector 400 
continues to update the road contour signal Sre according to the road judgement signal Sj until the lane detector 500 
detennines that the road contour signal Sre represents a correct road contour, and the road contour signal Sre is thus 

40 output only after it Is detemiined to represent a con-ect road contour. 

[0026] The local position detector 600 thus receives the road contour signal Sre from the lane detector 500, and a 
vehicle condition signal Sc input from the ECU 700. Based on the road contour signal Sre, which thus represents a 
reliable road contour, and the vehicle condition signal Sc indicative of the operating status of the motor vehicle, the 
local position detector 600 detects the moving or stationary position of the automobile AM, and thus generates a position 

4S detection signal Sp. It should be noted that a method and apparatus for generating an ECU control signal for controlling 
vehicle operation, and navigation and control signals for a navigation apparatus, based on this position detection signal 
Sp are disclosed In Japanese patent application HB-1 02892 (1996-102892; filed 24 April 1996), and United States 
Patent Application 08/637,417 (file 25 April 1996), both entitled and assigned to the same assignee as the present 
application. 

so [0027] A simplified side view of an automobile AM In which the digital imaging apparatus 1 00 of the present invention 
is Installed is shown in Fig. 4. The optical axis Ax of the digital imaging apparatus 1 00 is aligned with the direction of 
travel Z of the automobile AM on the road, but is oriented downward from the horizontal by a known number of degrees 
q. The view angle of the digital imaging apparatus 100 is also wide enough to capture the road area immediately below 
and in front of the automobile AM In the perspective image Vi. 

55 [0028] An exemplary perspective image VI of the forward direction as seen from the automobile AM and represented 
by the digital image signal Si captured by the digital imaging apparatus 100 is shown in Fig. 5. It is assumed in this 
image that the automobile AM is travelling in the left lane of a two lane road. The two lanes are defined by the three 
lane maricers Lm1 , Lm2, and Lm3. More specifically, the right lane is delineated by lane markers Lm3 and Lm2, and 
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the left lane by lane mai1<etB Lm2 and Lm1 . The lane markers are commonly applied using a high visibility paint such 
as white or yellow is commonly used. In the image shown in Fig. 5 the area at the bottom of the perspective Image Vi 
is an image of the road surface directly below the front of the automobile AM. 

[0029] It should also be noted that the perspective Image Vi captured by the digital imaging apparatus 1 00 is a typical 
5 video Image containing pixels of varying luminance, density, and hue, and that line drawing representations appearing 
as simple contour lines of objects In the viewing area are used In Fig. 5 because of simple drawing limitations. 
[0030] The prefen'ed embodiments of the present Invention are described next below with reference to the accom- 
panying figures. 

10 First Embodiment 

[0031] Fig. 2 Is a detailed block diagram of the spatial frequency separator 200, lane area detector 300, lane contour 
detector 400, lane detector 500, and local position detector 600 of the digital Imaging apparatus 1 00 shown in Fig. 1 . 
[0032] The spatial frequency separator 200 comprises a low frequency extractor 202 for extracting the low frequency 
IS component of the digital image signal Si and generating a low spatial frequency signal SL, and a high frequency ex- 
tractor 204 for extracting the high frequency component and generating a high spatial frequency signal SH. 
[00331 The low frequency extractor 202 uses the luminance data of the digital image signal Si to generate an image 
VL of the low spatial frequency component only, and in an exemplary embodiment of the invention a two-dimensional 
low pass filter. 

20 [0034] The high frequency extractor 204 similarly uses the luminance data of the digital image signal Si to generate 
an image VH of the high spatial frequency component only, and in an exemplary embodiment of the Invention a two- 
dimensional high pass filter. 

[0035] More specifically, the low frequency extractor 202 and high frequency extractor 204 separate the perspective 
image Vi captured by the digital imaging apparatus 100 into a low frequency Image VL and a high frequency image 

zs VH such that there is no overlap in the frequency components of the Images. 

[0036] The road area detector 300 comprises a lane brightness setter 302, a lane image extractor 304, and a lane 
region setter 306. The lane area detectorSOO is upstream of the lane detectorSOO, which relies on a region detennined 
by the area detector 300 and a contour extracted by contour detector 400. In this sense, it maybe more appropriate 
for the area detector 300 to be a road surface area detector and each of the internal components thereof to be road 

30 surface devices. It makes more sense for the lane contour detector 400 to be a "road contour detector" for contrast 
with the lane contour extractor 606. Hereinafter, uniformly used are the following tenns: road area detector 300, road 
brightness setter 302, road image extractor 304, road region setter 306. 

[0037] The road brightness setter 302 is connected to the low frequency extractor 202 of the spatial frequency sep- 
arator 200, and to the ECU 700, and receives respectively therefrom the low spatial frequency signal SL and the vehicle 
35 condition signal Sc. Based on the velocity signal Sv component of the vehicle condition signal Sc, the road brightness 
setter. 302 sets road brightness levels Bmin and Bmax, which define the luminance levels of the pixels In the area of 
the road in the low spatial frequency signal SL, and outputs a road brightness signal SBr indicative of the threshold 
luminance together with the low spatial frequency signal SL. 

[0038] Based on the low spatial frequency signal SL and road brightness signal SBr (Bmax) supplied from the road 
40 brightness setter 302, the road Image extractor 304 extracts the road region of the low spatial frequency signal SL, 
and generates a road extraction signal Srs. The road image extractor 304 thus supplies the low spatial frequency signal 
SL, road brightness signal SBr, and road extraction signal Srs to the road region setter 308. 
[0039] The road region setter 306 then determines what part of the low spatial frequency signal SL represents the 
region in which the road is located based on the road extraction signal Srs, and thereby generates the road region 
4S signal Srr supplied to the lane detector 500. 

[0040J The lane contour detector 400 comprises athreshold setter 402, contour extractor 404, and contour extraction 
region Ilmlter406. Connected to the high frequency extractor 204 of the spatial frequency separator 200, the threshold 
setter 402 sets the threshold value Eth extracted by edge detection processing the contour points of the image repre- 
senting the road in the high spatial frequency signal SH, and outputs a contour threshold signal Sth indicative of these 
50 threshold values Eth with the high spatial frequency signal SH. 

[0041] Based on the contour threshold signal Sth, the contour extractor 404 extracts the contour of the road image 
in the high spatial frequency signal SH, and thus generates the road contour signal Sre supplied to the lane detector 
500 and to the contour extraction region Iimiter406. 

[0042] Based on the road contour signal Sre, the contour extraction region limlter 406 produces a region limiting 
55 signal Sr restricting the region in the high spatial frequency signal SH to be used for road contour extraction, and feeds 
this region limiting signal Sr back to the high frequency extractor 204 of the spatial frequency separator 200. 
[0043] The high frequency extractor 204 thus extracts the high frequency component and generates the high spatial 
frequency signal SH from an area in the digital image signal Si restricted by the region limiting signal Sr. 
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[0044] The contour extractor 404 further receives a road judgement signal Sj as feedbacic from the lane detector 
500. If the road contour signal Sre is determined by the lane detector 500 to not represent the true contours of the lane 
or road, the road contour signal Sre is updated to use the contour edge adjacent to the contour line currently extracted 
as the lane contour as the updated contour line, and this updated road contour signal Sre Is then output to the lane 
5 detector 500. 

[0045] The lane detector 500 thus detennines whetherthe road contoursignal Sre represents the correct road contou r 
edge based on the road region signal Srr, and generates a road judgement signal Sj. In the exemplary embodiment 
of a lane detector 500 described herein, the road judgement signal Sj is preferably high when the correct road contour 
edge is extracted, and is otherwise low. In other words, when the road judgement signal Sj is low, the lane detector 
10 500 stops oulputting the road contour signal Sre, and causes the contour extractor 404 to update the road contour 
signal Sre. As a result, the lane detector 500 outputs the road contour signal Sre to the local position detector 600 only 
when the road judgement signal Sj is high. 

[0046] More specifically, therefore, the contour extractor 404 detects aserles of projected contour points representing 

candidates for the actual lane markers on the road using the high spatial frequency signal SH from the high frequency 
'5 extractor 204. This series of projected contour points Is output to the lane detector 500 as the road contour signal Sre. 
Using the road region data (I.e., the road region signal Srr) detected by the road Image extractor 304, and the projected 
contour points (i.e., the road contour signal Sre) detected by the contour extractor 404, the lane detector 500 then 
detects the contour points of the lane markers. 

[0047] The local position detector 600 comprises a coordinate converter 602, matching detector 604, lane contour 

20 extractor 606, and In-lane position detector 608. 

[0048] The coordinate converter 602 converts the front perspective image Vi to a birds'-eye image Vcc by applying 
a coordinate conversion process to the road contour signal Sre, and thus produces a birds'-eye contour signal Sec. 
[0049] Based on the birds'-eye contoursignal Sec supplied from the coordinate converter 602, the matching detector 
604 then detects whetherthe road surface for which a contour was detected Is straight or cun^ed to output a matching 

25 signal Sm Indicative of the matched road shape. 

[0050] Based on the matching signal Sm Input from the matching detector 604, the lane contourextractor606 extracts 
a contour of the lane currently occupied by the automobile AM to generate a lane contour signal S1 c. 
[0051] Next, based on the lane contoursignal S1 c input from the lane contour extractor 606 and the vehicle condition 
signal Sc supplied from the ECU 700, the in-lane position detector 608 detects the position of the automobile AM within 

3D the occupied lane to generate the position detection signal Sp output therefrom to the ECU 700. 

[0052] Note that the local position detector 600 generates the position detection signal Sp using a match detection 
operation based. In part, on the vehicle condition signal Sc supplied from the ECU 700. In an exemplary embodiment 
of the present Invention as described further below with reference to Figs. 20 to 22, the match detection operation of 
the matching detector 604 uses the steering signal SsT component of the vehicle condition signal Sc. The invention 

35 shall not be so limited, however, and other methods can be alternatively used. 

[0053] The operation of the essential parts of a local positioning apparatus LP according to a preferred embodiment 
of the present Invention is described next below with reference to the accompanying Figs. 5 through 25. The operation 
of the digital Imaging apparatus 1 00 has already been described above with reference to Fig. 5. The next step Is 
therefore to describe the edge extraction operation of the high frequency extractor204 of the spatial frequency separator 

40 200 with reference to Figs. 5 and 6. 

[0054] As described above, the high frequency extractor 204 extracts the high frequency component of the digital 
image signal Si by applying a Sobel filter or otherflltering process to the digital image signal Si. This filtering operation 
detects the edge pixels where there Is a sudden change in pixel density. The extractor 204 then generates a high 
spatial frequency signal SH representing an edge Image Vh. To reduce the amount of data that must be filtered for this 

4s extraction, the filtering operation is limited to a specifically limited part of the perspective Image VI. It is therefore possible 
to more quickly detect the perspective image edge Vh data, and generate the high spatial frequency signal SH. 
[0055] More specifically, the perspective Image VI (Fig. 5) is divided Into two parts, a top region St and a bottom 
region Sb, using a single horizontal line Ls. Note that the horizontal line Ls is aligned to the A-th vertical pixel counted 
from the bottom of the Image. The area of the top region St is therefore Ph x (PV - A) pixels, and the area of the bottom 

so region Sb is Ph x A pixels. The vertical position of this A-th pixel is preferably set to match the horizontal position of 
the vanishing point of perspective image Vi when the perspective image Vi is captured with the automobile AM on a 
level road. In the exemplary embodiment of the invention, pixel A Is set to the vertical position of a pixel representing 
an image positioned approximately fifty meters away from the automobile AM when the perspective Image Vi is captured 
directly by the digital imaging apparatus 100. 

55 [0056] Immediately after the local positioning operation of the present invention Is started, the high frequency ex- 
tractor 204 applies the above high spatial frequency component extraction (filtering) operation to the bottom region Sb 
only to extract the edge pixels only from and generate the high spatial frequency signal SH only for the bottom region 
Sb. This obviously means that the high spatial frequency signal SH only contains edge pixels extracted from the bottom 
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region Sb of Ph x A pixels. 

[0057] Tlie extracted edge image VI: represented by the high spatial frequency signal SH is shown in Fig. 6. The 
edge pixels of and near primarily the lane markers Lm1 , Lm2, and Lm3 are extracted from the bottom region Sb of the 
perspective image Vi (Fig. 5) as the edge pixels. Though not shown In Fig. 6, It should be obvious that edge pixels 
5 from objects unrelated to the lane mariners are also extracted. 

[0058] The threshold setter 402 thus receives the high spatial frequency signal SH comprising the extracted edge 
pixels from the low frequency extractor 202, and calculates the threshold value Eth for effectively extracting the edge 
pixels from the contours of the lane mariners Lm in the high spatial frequency signal SH using the following equation (1 ). 

Eth = C * Emax + (1 - C) * Emean (1 ). 

where Emax and Emean are the maximum density and the mean density values of the pixels on a particular horizontal 
line in the bottom region Sb of the perspective image Vi; and C is a constant where 0 < C < 1 . 

IS [0059] The contour extraction method of the contour extractor 404 Is described next referring next to Figs. 7 and 8. 
An edge Image Vh (SH) from which the contour extractor 404 detects contour lines is shown in Fig. 7. 
[0060] The contour extractor 404 scans the pixels in the bottom region Sb of the high spatial frequency signal SH 
using the threshold value Eth to extract the contour lines of the displayed lanes. This bottom region Sb Is further divided 
Into left and right bottom regions SbL and SbR by a vertical center line Lc. Note that the vertical center line Lc rises 

20 vertically from pixel Ph/2 on the bottom line of the image in this exemplary embodiment, but the Invention shall not be 
so limited. 

[0061] The contour extractor 404 compares the density of each pixel in the scanning area with the threshold value 
Eth, and extracts those pixels with a pixel density greater than or equal to the threshold value Eth as a contour pixel. 
The scanning order for this operation follows horizontal line Ls and worl<s to the left and right of the vertical center line 
25 Lc along left and right horizontal lines LhL and LhR. After scanning along horizontal line Ls, the contour extractor 404 
drops down a particular number of pixels along the vertical center line Lc, and again scans to right and left to obtain 
the contour pixels on that horizontal line. 

[0062] In exemplary practice, however, the contour extractor 404 scans first to the left or the right of the center line 
in the bottom region Sb to obtain the contour pixels on that side. For example, if the left side of the bottom region Sb 
30 Is scanned first, the contour extractor 404 scans top to bottom starting from horizontal line Ls at pixel A, scanning lines 

at an interval of PLh pixels downward to the bottom of the image, and scanning every pixel right to left from the vertical 
center line Lc along the left horizontal line LhL. After obtaining all contour pixels on the bottom left sector of the image 
In this sequence, the contour extractor 404 then returns, to the top horizontal line Ls to similarly scan the bottom right 
sector of the image in left to right sequence from the vertical center line Lc and obtain the contour pixels therein. 
35 [0063] Sl<ipping the horizontal scanning position at a particular Interval PLh results In extraction from an Integer K 
number of horizontal lines Lh equivalent to the absolute value of Pv/PLh. This reduces the number of calculations to 
be perfonned, and thereby increases the speed of contour pixel extraction. Note that the value of PLh is a natural 
number set in a pixel unit increment. 

[0064] When the first pixel with a particular density value is found on left or right horizontal lines LhL and LhR at a 
40 given vertical position, that pixel Pe Is defined as a contour pixel, and contour pixel scanning for that horizontal line is 
skipped. Scanning then advances directly to the next horizontal line PLh pixels down. 

[0065] In other words, contour pixels are detected at the Inside edges of the lane markers Lml and Lm2 on the left 
and right sides of the shown lane at the first horizontal line Lh counted down from the horizontal line Ls, as shown in 
Fig. 7. At the third and fourth lines, however, there is a gap between successive lane markers Lm2 dividing the right 

45 and left lanes, and contour pixel detection to the right side of the vertical center line Lc at these lines, or more specifically, 
at any vertical position between these lane markers Lm2, will fall to detect a contour pixel from any lane marker Lm2 
and may detect a contour pixel from the lane marker Lm3 at the edge of the adjacent lane. The edges of markings 
other than the lane markers but causing a similar sudden change in pixel density, e.g., soil or even imaging flaws, will 
also be detected as contour pixels. Such contour pixels are, however, irrelevant to detecting the vehicle lanes, and are 

50 therefore simply a noise component insofar as the position detection operation of the present invention is concerned. 
As a result, these noise components are removed by the lane detector 500 as described in further detail below. 
[0066] An extracted contour image Vre represented by the road contour signal Sre obtained as described above 
from the image In Fig. 7 is shown In Fig. 8. The detected contour pixels of the lane markers Lm1 Lm2, and Lm3 are 
shown as contour signals ScL, ScR, and ScR' where ScR' is noise data. Contour pixel signal ScL' represents noise 

55 data extracted due to some other flaw, shadow, or marking apparent in the image, 

[0067] The method of limiting the detection area by means of the contour extraction region limiter 406 is described 
next below with reference to Figs. 9 and 10. As shown In Fig. 9, the contour extraction region limiter 406 defines a 
limited region RsL and RSR of a known width Wr to the right and left sides of the extracted contour signals ScL and 
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ScR based on the road contour signal Sre. The region limiting signal Sr Identifies these limited regions RsL and RsR, 

and Is output to the high frequency extractor 204 of the spatia! frequency separator 200. However, when the contour 
extractor 404 is unable to extract the lane contour and an en-or signal. See (not shown In the figures) in place of the 
road contour signal Sre, the extraction region Is not limited. 

[0068] The width Wr of the limited region Is detemiined with respect to the amount of movement in the x-axls (hori- 
zontal) direction of the lane markets In the Image as a result of lateral movement of the automobile AM . More specifically, 
Wr is the distance that lane marker Lm1 can move in the perspective Image VI in one system cycle, I.e., 33 mS in an 
exemplary embodiment of the invention, it therefore follows that the limit width Wr increases towards the bottom of the 

image. . . , „ 

[0069] As shown in Fig. 1 0, the high frequency extractor 204 limits the area for edge extraction in the bottom region 
Sb of the perepective image Vi to the limited regions RsR and RsL based on the region limiting signal Sr. In addition 
to reducing the amount of data to be processed and therefore Increasing the processing speed, this process also 
suppresses noise components from outside the lane contours, and therefore Improves the lane tracking abllily. 
[0070] When a noise component is sufficiently removed from the limited regions RsR and RsL, as is the contour line 
Scr" of the opposing lane marker Lm3 described above, the noise component is removed by the contour extraction 
region Ilmiter406. However, when a contour noise component is near a lane contour, as is the contour line ScL in Fig. 
8, the noise component will be within the limited region Rsr. In this case the noise component cannot be removed by 
the contour extraction region Iimiter406. As a result, these noise components must be removed by the lane detector 
500, as will be described later below. 

[0071 ] The method of generating a low spatial frequency image by means of the low frequency extractor 202 of the 
spatial frequency separator 200 is described next below with reference to Fig. 1 1 . 

[0072] The low frequency extractor 202 generates the low spatial frequency signal SL by low-pass filtering the digital 
image signal SI. The subject of the perspective Image VI appears as a fuzzy grouping of pixels of various luminance 
levels in the low frequency image VL represented by the low spatial frequency signal SL. The low frequency image VL 
represented by the low spatial frequency signal SL Is thus a luminance Image. To reduce the processing load of the. 
low spatial frequency signal SL generating operation, the luminance pixels are therefore extracted from a limited part 
of the perspective Image VI, specifically, the Ph x A pixel bottom region Sb. This is similar to the method of generating 
the high spatial frequency signal SH described above. 

[0073] The luminance of the pixels in the road region of the low frequency Image VL tends to Increase with the 
distance from the automobile AM, i.e., the distance from the digital Imaging apparatus 100. The same is true of the 
luminance level of pixels in image areas corresponding to the road shoulder, drainage channels, and guard rails. In 
the Image in Fig. 11 , the pixels at vertical pixel level A, i.e., along horizontal line Ls, have the highest luminance level 
of any pixels associated with the same object, and the pixels toward the bottom of the image have the lowest luminance 
level. 

[0074] The luminance level of pixels at the same vertical height but associated with different objects will, however, 
vary from object to object. More specifically, the luminance will vaiy according to the reflectance of the subject material, 
and the angle of incidence of light from the light source. The above described relationship between luminance and 
distance, i.e., that luminance increases with distance, is, however, retained. 

[0075] The low frequency image VL is thus produced only from the bottom region Sb of the perspective image VI 
obtained by the digital imaging apparatus 100 imaging primarily the area directly in front of the vehicle. Furthemiore, 
a majority of the pixels in the low frequency image VL represent the road surface, and this tendency is therefore 
substantially applicable to all pixels in the low frequency image VL. The pixels forthe road surface detected along the 
horizontal line Ls atthe farthest point from the digital Imaging apparatus 1 00 in the low frequency image VL are therefore 
defined as the farthest road pixels Pf , and the luminance level of those pixels is defined as maximum luminance Bmax. 
The pixels for the road surface detected at the bottom of the image nearest the digital imaging apparatus 1 00 in the 
low frequency image VL are therefore defined as the nearest road pixels Pn. and the luminance level of those pixels 
as minimum luminance Bmln. 

[0076] Note that the location of the farthest road pixels Pf and nearest road pixels Pn can be freely set as required. 
[0077] A histogram of all pixels in the low frequency image VL is shown in Fig. 12 with luminance levels plotted on 
the horizontal axis and the number of pixels with a given luminance level plotted on the vertical axis. As will be known 
from Fig. 12, a majority of the pixels in the low frequency image VL are road surface pixels with luminance between 
minimum luminance Bmln and maximum luminance Bmax. The luminance range between minimum luminance Bmln 
and maximum luminance Bmax is therefore called the road luminance range Br, and the pixels in this road luminance 
range Br are called road pixels Pr 

[0078] When the digital imaging apparatus 1 00 is installed at substantially the front center of the automobile AM as 
described with reference to Fig. 4, the positions of the farthest road pixels Pf and nearest road pixels Pn are set at 
appropriate vertical positions Vpf and Vpn in the image on the vertical center line Lc. The road surface occupying the 
area between Vpf and Vpn is then indicated by the road pixels Pr. The position of Vpf can therefore be thought of as 
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the farthest distance at which the road surface is detected, and. the position of Vpn as the nearest distance at which 

the road surface is detected. 

[00791 The position of the farthest road pixels Pf is set appropriately on the farthest road detection distance Vpf 
according to the position at which the digital imaging apparatus .1 00 Is mounted on the automobile AM, and the position 

5 of the nearest road pixels Pn is liltewlse set appropriately on the nearest road detection distance Vpn. 

[0080] The farthest road detection distance Vpf and nearest road detection distance Vpn can be set according to 
the vehicle speed and the processing capacity (speed) of the local positioning apparatus LP as described further below. 
[0081 ] The relationship between the luminance of only road pixels in the low frequency image VL and the distance 
of the road surface from which the pixels are captured is shown in Fig. 1 3. The values shown in Fig. 13 are for pixels 

10 in an image experimentally obtained while driving on a straight section of highway on a sunny day with the digital 
imaging apparatus 1 00 mounted on the front of the automobile AlW. The horizontal axis indicates the horizontal scan ning 
position (Pv) in the low frequency image VL, i.e., the relative distance D of each pixel from the digital imaging apparatus 
1 GO, and the vertical axis indicates the luminance B at that distance. That the distance - luminance relationship is not 
a flat line but has a right ascending slope is a result of lane markings, soiling, and color and luminance spots caused 

is by reflectance from obstructions on the road. As already described above, the area represented by a single pixel 
increases as the horizontal scanning position is elevated, meaning that the scanning position becomes more distant 
from the image pickup 1 00. However, the relationship between imaging distance and pixel luminance can be expressed 
by equation (2) below. 

^° B = A*D + C (2) 

[0082] Note that values of A = 0.41 28 and D = 1 08.56 have been experimentally obtained on straight lanes of an 

expressway. 

25 [0083] By correctly setting C and C" so that C > C > C" , and obtaining parallel lines of: 

B = A*D + C' (3). 



30 and 

B = A*D + C" (4), 

35 above and below the line represented by equation (2), the range of luminance values of pixels at a particular scanning 

position can be obtained. 

[0084] The method whereby the road brightness setter 302 detemnines the luminance level of the road surface Is 
described next below with reference to Fig. 14. Based on the relationship between the imaging distance and pixel 
luminance described with reference to Figs. 11 to 13, the farthest road pixels Pf and nearest road pixels Pn in the low 

40 spatial frequency signal SL are set to the pixel positions corresponding to the distance defining the road area to be 
used for positioning detection. The pixel luminance atthe farthest road pixels Pf is then defined as maximum luminance 
Bmax, and the pixel luminance at the nearest road pixels Pn Is defined as the minimum luminance Bmin to generate 
the road brightness signal SBr. Note that the road brightness signal SBr may be set to have luminance levels in the 
range defined by equations (3) and (4) above. The locations of the farthest road pixels Pf and the nearest road pixels 

45 Pn In the low frequency image VL are expressed as Pf(xPf ,yPf) and Pn(xPn,yPn) using the coordinate system shown 
in Fig. 3. 

[0085] By setting the position of the farthest road pixels R from which the maximum luminance Bmax Is detected to 

an appropriate position according to the processing speed Tp of the local positioning apparatus LP and the speed of 
the automobile AlVl, high precision lane detection is possible based on the operating conditions of the automobile AlVI 

50 on which the local positioning apparatus LP is installed. 

[0086] For example, by moving the farthest road pixels Pf vertically upward in the low frequency image VL according 
to the velocity signal Sv of the automobile AM, the detection precision at points far from the automobile PM can be 
maintained even when the automobile AlVl is travelling at a high rate of speed. When the automobile AM Is travelling 
slowly, lane detection precision can be improved by moving the position of the farthest road pixels Pf closer. Because 

55 the vehicle will pass the detection point while the detection process is in progress when the vehicle is travelling fast 
and the farthest road pixels Pf is close, it is preferable to detect the maximum luminance Bmax at a suitable distance 
from the vehicle based on the velocity signal Sv and the processing capacity of the local positioning apparatus LP. 
[0087] When the automobile AM is travelling on a straight road, the farthest road pixels Pf can be set on the vertical 
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center line Lc of the low frequency Image VL However, when the automobile AM Is on a curving road, setting the 
farthest road pixels Pf on the vertical center line Lc may result In the maximum luminance Bmax being extracted from 
a guard rail or other non-road subject within the low frequency image VL. 

[0088] In such cases, it is necessary to estimate the curvature of the road from the steering angle of the automobile 
5 AM so that only pixels within the image area of the road surface in the low frequency image VL are used for extracting 
the maximum luminance Bmax. This can be accomplished by moving the farthest road pixels Pf horizontally within the 
low frequency image VL according to the steering signal Ss so that the farthest road pixels Pf are within the road 
surface area. 

[0089] By moving the position of the farthest road pixels Pf freely horizontally and vertically within the low frequency 

10 image VL based on the velocity signal Sv and steering signal Ss, the farthest road pixels Pf can be adjusted to the 
road image area. Note that a value other than the velocity signal Sv can be alternatively used. Including a distance 
value (vertical image position value) entered by the user. Data on the direction of travel (straight or curved) supplied 
by a navigation system based, for example, on map data, can also be used in place of the steering signal Ss. More 
specifically, the pixels of the subject area can be reliably captured by moving or fixing the positions of the farthest road 

IS pixels Pf and nearest road pixels Pn in the low frequency image VL as necessary. 

[0090] The method whereby the road image extractor 304 extracts an Image of the road surface is described next 
below with reference to Figs. 15 to 1 7. The road image extractor 304 scans the pixels in the bottom region Sb of the 
low spatial frequency signal SL based on the pixel luminance relationship to the scanning position shown in Fig. 13 
and defined by equations 2, 3, and 4 to extract the pixels in the road image area. The road image contours extracted 

20 by the road image extractor 304 are similar to the contour lines of the lanes obtained by the contour extractor 404 and 
described with reference to Fig. 1 2. The road image extractor 304, however, extracts the pixels with a luminance level 
between maximum luminance Bmax and minimum luminance Bmin, i.e., pixels satisfying equations 2, 3, and 4 at each 
horizontal scanning position Pv, rather than using an edge extraction method as does the contour extractor 404. 
[0091] Region pixel sets Elm, Erm, and Emi' corresponding to the contour pixels ScL, ScR, and ScR' in Fig. 8 are 

SB obtained at the inside circumference part of lane markers Lml , Lm2, and Lm3. It should be noted that lines representing 
the inside edges of the lane marl<ers Lm1 , Lm2, and Lm3 can be obtained as necessary based on the region pixel sets 
Elm, Erni, and Erm' using a least squares method or other suitable method such as the Hough conversion described 
further below with reference to Figs. 20 and 22. An example of Hough conversion of pixel set Elm is shown in Fig. 1 6, 
and an example of a linear approximation of the road surface is shown in Fig. 17. Note that linear approximation 

30 effectively removes the Inside edge part set Emi' of the lane marker Lm3 of the adjacent lane. 

[0092] The method whereby the road region setter 306 defines the road surface area is described next below with 
reference to Fig. 1 8. 

[0093] The road region setter 306 sets road edge areas AeL and AeR based on the road extraction signal Srs by 
adding a known horizontal width Wa to both right and left sides of the extracted inside edge areas Elm and Em. The 

35 road region signal Srr output to the lane detector 500 Is Indicative of these road edge areas AeL and AeR. When the 
road region setter 306 cannot extract the road area and the error signal See is output In place of the road extraction 
signal Srs, the road edge areas AeL and AeR are not set Note that while the operation of the road Image extractor 
304 is similar to that of the contour extraction region limiter 406, the limit width Wa can be an appropriately defined 
value based on the values C, C, and C" shown in equations 2 to 4 above. 

40 [0094] Using the low spatial frequency signal SL from the low frequency extractor 202, the road image extractor 304 
divides the low frequency image, or luminance Image, VL Into areas of pixels having a particular relationship based 
on the luminance distribution. Certain road knowledge Is then referenced and applied to these divided regions to detect 
the road area. 

[0095] The "road knowledge" used herein includes knowing that the area containing the farthest road pixels Pf and 
IS nearest road pixels Pn in the perspective image Vi captured by the digital imaging apparatus 100 is within the road 
area. Other knowledge includes knowing that the vanishing point at which the right and left lane markers intersect does 
not change suddenly, and that the pixel area of the road surface is substantially constant. The area containing the 
nearest road pixels Pn and the adjacent area can therefore be defined as the road area. 
[0096] The operation of the lane detector 500 Is described next. 
50 [0097] The lane detector 500 determines whether the contour lines ScR and ScL described by the road contour 
signal Sre from the lane contour detector 400 (Fig. 8) are within the road edge areas AeR and AeL described by the 
road region signal Srr. To accomplish this the lane detector 500 detenmines whether the ScR value is within AeR+Wa 
at the same horizontal scanning line position (distance). If ScR is within this range, the road contour signal Sre is 
detemnined to represent a correct road contour. The road judgement signal Sj Is therefore output low, and the road 
55 contour signal Sre is passed through to the local position detector 500. 

[0098] However, if ScR is outside the range Aef^ttWa, the road contour signal Sre is determined not to represent a 
correct road contour, but to be within a noise contour component. The road judgement signal Sj Is therefore output 
high, and the road contour signal Sre is not output 
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[0099] It is tlierefore possible as described above to produce a iow frequency image and a higli frequency image 
Witt! no spatial frequency overlap between the images from tlie same luminance image data, and separately detect 
from these images the road area and lane mari<er contour points. As a result, lane markers can be detected with high 
precision. 

[0100] The coordinate conversion principle used by the coordinate converter 602 to convert the extracted contour 
image Vre is described briefly below with reference to Figs. 3 and 4. 

[0101] As described above, the coordinate convertor 502 converts the coordinate system of the road contour signal 
Sre to convert the extracted contour Image Vre represented by the road contour signal Sre to a birds'-eye image Vcc. 
However, the perspective representation of objects in the extracted contour Image Vre results In shape distortion that 
Increases with the distance from the digital imaging apparatus 1 00. This means that, for example, when there are two 
identical objects with one placed at a greater distance from the digital imaging apparatus 1 00, the farther object will 
appear smaller than the nearer object. This shape distortion Increases with the distance from the digital imaging ap- 
paratus 1 00 or automobile AM. 

[0102] The coordinate system of perspective Images Vi, Vh, and Vre Is shown In Fig. 3. The birds'-eye view coordinate 
system referenced to the automobile AM is shown in Fig. 4. The x-axis of the image coordinate system (x,y) Is parallel 
to and oriented In the same direction as the X-axis In the coordinate system of the birds'-eye image Vcc, and the y- 
axis is inclined to the Y-axIs q degrees because the optical axis Ax of the digital imaging apparatus 100 is inclined q 
degrees from level. The X-axis and Z-axIs therefore define a level plane Pd that Is parallel to the travelling surface, I. 
e., road surface, of the automobile AM. The Y-axis and y-axis are aligned, and the X-axis is peipendicular to the surface 
plane of Fig. 4. The origin O matches the original of the coordinate system of the extracted contour image Vre, The 
optical axis Ax of the digital Imaging apparatus 100 passes through the origin 0. 

[0103] The coordinates of the extracted contour Image Vre can be converted to the coordinates of the birds'-eye 
image Vcc using the following equations. 

X = (x/F) ■ (Z cos e - Y sin 6) (5). 
Z = Y(Fcose + Yslne) 

and 

Y = -H (7). 

where F is the focal distance of the optical lens of the digital imaging apparatus 1 00; 6 is the angle between the Z-axis 
(horizontal axis) and the optical axis Ax of the lens; and H is the distance from the road surface to the origin of the 
optical axis of optical lens. The values of 6 and H are preferably set to obtain a perspective image VI of the area to the 
front of the automobile AM as described with reference to Rg. 5. In an exemplary embodiment of the present Invention, 
e Is 6 degrees, and H is 1 .2 meters. 

[0104] The birds'-eye contour signal Sec Is then obtained by converting the coordinates of the road contour signal 
Sre by converting the distances in the extracted contour image Vre using the above equations 5 to 7. Horizontal dis- 
tances In this birds'-eye contour signal Sec are accurately represented irrespective of the Z-axis direction from the 
digital imaging apparatus 100. More precisely, the coordinate-converted image Vcc is not a birds'-eye image, but is a 
plan view of the road surface captured from a plane parallel to the road surface. Furthemiore, image Vcc always 
represents the road surface as a flat surface even when, for example, there are low rises or obstructions in the road. 
This error/difference (low rises or obstructions), however, does not degrade the detection precision of the local posi- 
tioning apparatus according to the present Invention. This is because the positioning detection operation of the present 
invention uses a relatively near distance of only approximately fifty meters fonward as the farthest forward point used 
for detection and control, and bumps and obstructions on the road surface can therefore be ignored. 
[0105] |fafullbirds'-eyeimageoftheroadsurface,l.e.,aplanvlewlmageoftheroadsurfacecapturedperpendiculariy 
to a horizontal plane perpendicularto the vertical line with respecttothe gravitational axis Is required, It can be obtained 
by various methods. One such method is to not fix the optical axis Ax of the digital imaging apparatus 100 to the 
automobile AM, and use a gyroscope or other automatic attitude control device to always obtain the image Vi from a 
constant attitude to the horizontal. 

[01 06] For reference, reverse coordinate conversion from a birds'-eye view coordinate system to a perspective view 
coordinate system can be achieved using equations B and 9 below. 
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x=FX/(Zcose + Hsin 6) 



(8) 



2 = F{H cos 6 + Z sin G) / (Z cos 6 - H sin 9) (9) 

5 

[0107] The contour image Vcc represented by the coordinate-converted birds'-eye contour signal Sec is shown In 
Fig. 19. The lane marlcers Lm1 , Lm2, and Lm3 are represented by corresponding edge contours ScL, ScR, and ScR". 
As will be l<nown by comparison with the extracted contour image Vre in Fig. 8, the edge contours ScL, ScR, and ScR' 
10 are parallel throughout Fig. 1 9, and represent the actual lanes on the road. In other words, the coordinate-converted 
bIrds'-eye contour signal Sec contains correct dimension Infomnation for the subject 

[0108] Based on the coordinate-converted bIrds'-eye contour signal Sec, the matching detector 604 connected to 
the coordinate converter 602 obtains line segments or arcs matching the edge contours ScL, ScR, and ScR' by applying 

the following equations. 

is [0109] The first step is a Hough conversion of the pixel data for the edge contours ScL, ScR, and ScR' of the lane 
markets In the bIrds'-eye contour signal Sec. This Is accomplished by applying equation (1 0) below separately to the 
corresponding contour lines. 

^ p = Xcosi|i-i-Zsin(|) (10) 

where p Is the distance between the origin O and the pixel in the Z-X coordinate system, and ^ is the angle between 
the X-axis and a line joining the origin O and a particular pixel. The following equation (11) is then obtained after 
calculating equation (10). 

X =(p -Zsin(|i)/cos(|) (11) 

[01 1 0] A group of curves is obtained for each contour line by scanning the birds'-eye contour signal Sec and con- 
so verting the contour line data to parametric space based on the above Hough conversion. The method of detemiining 
whether a contour line is a straight line segment or arc based on this group of curves is described below with reference 
to Figs. 10 to 22. 

[0111] Typical patterns obtained when the Hough-converted line data Is a straight line Is shown in Fig. 20. Note that 
Ideally each of the curves intersect at a single point Cp. As a practical matter, however, the lane marl<ers Lm and 

35 contour lines are not usually completely straight lines, and convengence at a single point as shown in Rg. 20 thus rarely 
occurs. This problem can be resolved by noting that the curves tend to Intersect at the same point, and investigating 
the frequency Fc that each curve intersects at each point (pixel) in the parametric space, if this frequency Fc is greater 
than a particular threshold value Eth, that point is defined as the unique point of intersection Cp. The corresponding 
contour line is then defined as a straight line, and the appropriate equation for a line is obtained. 

40 [0112] Typical patterns obtained when the Hough-converted line data is a curve is shown in Figs. 21 and 22. As 
shown in these figures, there is neither no point of intersection (Fig. 21 ), or a group of curves intersect at a plurality of 
points Cpn (Cp1 to Cp6) as shown in Fig. 22. Note that n in this ease Is an integer value. The frequency of intersection 
Fc Is again obtained. If there is no point where anew frequency threshold Fth' is exceeded, the group of cun/es is 
detemiine to not Intersect, and the con-esponding contour line Is detennined to be a curve (arc). The appropriate 

45 equation for a curve Is then obtained. 

[01 13] it should be noted that these equations make it possible to detemiine particular dimensional features of the 
road, i.e., the lane that the automobile MH is either following or stopped in. 

[0114] As shown in Fig. 2, the lane contour extractor 60B is connected to the matching detector 604, and receives 
therefrom the matching signal Sm. Note that the matching signal Sm also contains contour line data ScR', which is 

50 obtained from noise components as described above. 

[01 1 5] The lane contour extractor 606 compares the dimensional features data contained in the matching signal Sm 
with particular predetemilned data such as the vehicle width, lane width, and the pattern of the center line lane marker 
Lm2 to remove contour data ScR' that is not appropriate to the target lane. The remaining contour line data ScR and 
ScL corresponding to the cun-ent lane Is output as the matching signal Sm. If there Is no data corresponding to the 

55 lane in the matching signal Sm, an error signal See (not shown in the figures) is output. 

[0116] The filtering operation of the lane contour extractor 606 is described next with reference to Fig. 1 9. 
[0117] This bIrds'-eye image Vcc contains three contour lines ScL, ScR, and ScR'. Counting down from the top of 
the image, the left two contour lines ScL and ScR are a contour line pair describing a single lane Lm down to the second 
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pixels. The center line lane marlcer Lm2 in this Image, however, is a divided line, and the contour line ScR' at the far 
side of the adjacent lane Is therefore extracted from the birds'-eye contour signal Sec at pixels 3 to 5. The distance 
between ScL and ScR' in this case is obviously greater than the width of a single lane, but it is still not known at this 
point whether contour line ScL or ScR' represents noise. 

5 [0118] From pixel 6 to 11 , however, contour lines ScL and ScR are obviously paired. The left contour line ScL is also 
determined by the matching detector 604 to match a single arc ScLm, and pixels 3 to 5 furthermore corresponding to 
lane marker Lm1 . The three pixels from pixel 3 to 5 in contour, line ScR' are therefore ignored as noise. The two contour 
lines ScL and ScR are therefore extracted as a correct contour line pair. The matching contour lines ScLm and ScRm 
are thus selected, and output with the information of selected contour lines as the lane contour signal S1 c 

10 [0119] Combined with the road area evaluation result output as road contour signal Sre by the lane detector 500, 
the lane contour extractor 606 thus effectively eliminates noise components, such as other markings and soiling on 
the road, that are unrelated to lane definition from the lane contour signal S1 c. 

[01 20] The extracted lane contour Image Vie represented by the road contour signal Sle after noise removal by the 

lane contour extractor 606 is shown in Fig. 23. 

IS [0121] The operation of the in-iane position detector 608 Is described next with reference to Fig. 23. 

[0122] Using the lane contour signal S1c, the in-lane position detector 608 obtains the relationship between a circle 
or line tangential to or intersecting the right and left contour lines ScLm and ScRm to obtain the points of . intersection 
HL and HR at Z = 0. The coordinates for the midpoint of this line segment, i.e., the coordinates (HC,0) of the line 
segment HL-HR, are then obtained. Note that the camera is located at (0,0). Therefore, if the digital imaging apparatus 

20 1 00 is mounted at the widthwise center of the automobile AM, the position of the automobile AM relative to the lane 
can be expressed as the lateral displacement -HC from the lane center. 

[0123] The overall operation of a local positioning apparatus LP according to an exemplary embodiment of the present 
invention Is described next below with reference to the flow chart in Fig. 24. 

[0124] Operation starts after a digital image signal Si is generated by means of the digital imaging apparatus 1 00 
2S capturing a picture of the view to the front of the automobile AM (block #1 ). 

[0125] The spatial frequency separator 200 then extracts the low spatial frequency signal SL and high spatial fre- 
quency signal SH from the digital Image signal Si (block #3). 

[0126] The threshold setter 402 then sets a threshold value Eth based on the edge pixel density In the high spatial 
frequency signal SH, and generates the contour threshold signal Sth (block #5). 
30 [0127] BasedonthecontourthresholdsignalSth,thecontourextractor404extractsthelanemarkeroontours using 
the threshold value Eth derived from the high spatial frequency signal SH, and outputs the road contour signal Sre 

(block #7). 

[0128] Whether an en-or signal See Is output in place of the road contour signal Sre is then detennined (block #9). 
If the error signal See is output, contour lines have not been extracted. A YES Is therefore returned and the procedure 

35 loops back to block #1 . If NO Is returned, the procedure advances. 

[01 29] Based on the road contour signal Sre, the contour extraction region iimiter 406 then defines the area in the 
high spatial frequency signal SH to be used for contour extraction, and outputs a region limiting signal Srto the high 
frequency extractor 204 (block #1 1 ). This feedback loop (see Fig. 2) causes the high frequency BXtractor204 to extract 
the high frequency component in the area defined by the region limiting signal Sr within the digital Image signal Si at 

40 block #3 above, and generate the high spatial frequency signal SH from this limited region. 

[0130] The road brightness setter 302 then sets a road surface luminance area (Bmax, Bmin) in the low spatial 
frequency signal SL resulting from block #3, and outputs a road brightness signal SBr (block #13). 
[0131] Based on the road brightness signal SBr, the road image extractor 304 then extracts the road surface image 
from the low spatial frequency signal SL to generate the road extraction signal Srs (block #15). 

4S [0132] Next, based on the road brightness signal SBr and road extraction signal Srs, the road region setter 306 
generates a road region signal Srr describing the road edge areas AeR and AeL (block #17). 
[0133] Whether the error signal See is output In place of the road region signal Srr is then detected (block #1 9). If 
the road region has not been detected, YES Is returned and the procedure loops back to block #1 . If NO is returned, 
the procedure advances. 

so [0134] The lane, detector 500 then detemiines whether the extracted contour lines described by the road contour 
signal Sre are valid based on the road region signal Srr (block #21). If the result is NO, the contour extractor 404 is 
driven again by means of the road judgement signal Sj to update the road contour signal Sre. The sequence from block 
#7 to block #21 Is thus repeated until valid contour lines are extracted. The road contour signal Sre is passed to the 
local position detector 600 only once the contour lines are detennined to be valid (con-ect). 

SB [0135] The coordinate converter 602 then coordinate converts the extracted contour image Vre of the road contour 
signal Sre to a birds'-eye Image Vccto generate a birds'-eye contour signal Sec (block #23). 
[0136] The matching detector 604 then uses a Hough conversion or other appropriate conversion method to match 
an equation describing the Intemal contour lines of the birds'-eye contour signal Sec to straight line segments or arcs 
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(block #25). The matching detector 604 also outputs a matching signal Sm representing the matching line segmente 
or arcs. 

[0137] Whether an error signal See is output in place of the matching signal Sm Is then determined {block #27). If 
there are no matching line or arc segments, YES is returned and the procedure loops backto block #1 . If NO is returned, 

s the procedure advances. 

[0138] Dimensional features of the matching contour lines indicated by the matching signal Sm are then compared 
with the dimensional features of the automobile AM by the lane contour extractor 606 (block #29). The lane contour 
extractor 606 thus extracts a pair of lane contour lines describing the lane currently occupied by the vehicle, and 
generates a lane contour signal S1 c. 

10 [0139] The current position of the automobile AM is then detected by the in-lane position detector 60B based on the 
lane contour signal S1c, and a position detection signal Sp is thus output (block #31). 

[01 40] Note that, if the error signal See is detected in blocks #9, #1 9, or #27 above, the procedure is returned to the 
beginning (block #1) to achieve more precise lane detection and position detection, by executing the procedure for 
each intervening block based on the high spatial frequency signal SH and low spatial frequency signal SL extracted 
IS at the same time. Depending on the desired precision, however, blocks #9, #1 9, or #27 can be omitted without sub- 
stantially affecting the results achieved by the present invention. 

Second embodiment 

20 [0141] A local positioning apparatus according to a second exemplary embodiment of the present invention is de- 
scribed next below with reference to Figs. 26 to 45. Fig. 26 is a block diagram of a road area extraction apparatus 
according to the present embodiment of the invention, which comprises a digital image pickup 1 1 , brightness histogram 
generator 1 2, lane brightness separator 1 3, and lane polygon detector 1 4. 

[0142] The digital image pickup 11 is typically a video camera used for capturing an Image of the road in front of the 
25 vehicle. If the digital image pickup 11 is mounted at approximately the center front of the vehicle, the bottom center 
portion of the captured image will necessarily contain the road surface, as is described further below. 
[0143] The brightness histogram generator 1 2 produces a brightness histogram, as shown in Fig. 27, from the input 
road image data. Using this brightness histogram, the brightness histogram generator 12 also detects the vehicle 
position in the image, i.e., detects the luminance at the bottom middle part of the image. 
30 [0144] Using the histogram, the lane brightness separator 13 detects the change in brightness to the right and left 
sides of the luminance at the vehicle position in the image. By detecting valleys in the brightness level, the lane bright- 
ness separator 13 finds the luminance range corresponding to the lane area, and extracts the pixels in that range as 
lane area candidate pixels. 

[0145] Referencing the extracted lane area candidate pixels, the lane polygon detector 14 finds the edges of the 
35 corresponding pixel area to the right and left sides of the bottom scanning line of the image, it then detects the midpoint 

between these side edges, and then repeats the edge pixel detection process at another scanning line higher In the 
image. The result of this process is a polygon that is detected as the road area. 
[0146] This process is described more specifically below with reference to the flow chart in Fig. 28. 
[01 47] The first step is to generate a brightness histogram from the luminance data captured by the digital imaging 
40 pickup 11 (step SI). 

[01 48] Using the brightness bO at the bottom center of the image, the position In the histogram con-esponding to the 
road surface directly below the vehicle is obtained (step S2). A valley on the left side is detected first. 
[0149] Mean brightness values b^ean"! b^^2 are then obtained for the overlapping brightness regions bo - b, 
S b < bo and bo - 3bi/2 < b < bo - 6^/2 where b is a brightness parameter and b^ is a constant brightness level (step S3). 
45 [0150] The equation b^eana = b„ean2 - bmeani 's then calculated (step S4), and the sign of b^^aifi is then determined 
(step S5). A valley is detected by detennining when the sign changes from negative to positive. 
[0151] If the sign has not changed from negative to positive, bo = bo - b-^/Z (step S6), and the procedure loops back 
to step S3 (step S6). This process is repeated until a valley is detected to the left side. 

[0152] A similar process is then executed to detect a valley on the right. The first step (step 87) is to reinitialize bo 
50 to the value detected in S2, and then calculate b^^^S and b^^g„A for the regions bo < b Sbo + b, and bo + bi/2 < b S 
bo + 3bi/2. The equation b^eanb = b^em^ - bmean3 is then obtained (step S8). 

[0153] A valley is detected by determining when the sign changes from negative to positive. If the sign has not 
changed from negative to positive, bg = bo + b^/2 (step SI 0), and the procedure loops backto step 87 (step S9) . This 
process is repeated until a valley is detected to the right side. 
55 [01 54] The pixels in the brightness region between the detected valleys are then detected as the lane area candidate 
region of the image (step S11). 

[01 55] Referencing the extracted lane area candidate pixels, the edges of the pixel areas to the right and left sides 
of the bottom scanning line of the image are detected as shown in Fig. 29. It then detects the midpoint between these 
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side edges, and then repeats the edge pixel detection process at another scanning line higher In the Image. The result 
of this process is a polygon as shown in Fig. 29 detected as the road area. 

[0156] As described above, the road area extraction apparatus of the present embodiment detects an area containing 
the luminance value directly below the vehicle in the brightness histogram. Using the image area corresponding to this 
5 brightness value, the edges of the road area are then detected using a center-outward pixel comparison method. The 
road edges thus extracted are an accurate representation of the road area In that Image. 

[0157] Af irst aitemative of second embodiment of a road area extraction apparatus according to the present Invention 
is described next below with reference to Figs. 30 and 31. This road area extraction apparatus comprises a digital 
image pickup 21 , brightness distribution pattern detector 22, and lane image separator 23. 

10 [0158] The digital image pici<up 21 is typically a video camera used for capturing an image of the road in front of the 
vehicle. If the digital image pickup 21 is mounted at approximately the center front of the vehicle, the bottom center R 
portion of the captured image will necessarily contain the road surface, as is described further below. 
[0159] The brightness distribution pattern detector 22 samples a defined region Sq from the bottom middle of the 
road image data as shown in Fig. 31, and determines the brightness distribution a in this area. 

15 [01 60] The lane Image separator 23 then divides the lane Image and compares the brightness distribution b of each 
image segment with the brightness distribution pattern a obtained by the brightness distribution pattern detector 22 as 
shown in Fig, 32. If the pattems match, the image area is determined to be a road area. 

[0161] The operation of this road area extraction apparatus Is described below with reference to the flow chart In 

Fig. 33. 

20 [0162] The brightness distribution pattern of the defined region Sq at the bottom middle of the road image data is 
obtained by caiculating brightness mean m and distribution s in region Sq. The maximum logarithmic probability 1 is 
then calculated as follows where n is the number of data points in the region Sq. 

1 a2) = - (n/2) iog27ia2 - n/2 (12) 

[0163] The road image is then separated into N x N (the number of data points In this region), the brightness distri- 
bution pattern of an area Si different from region SO is obtained by calculating brightness mean [li and distribution oi 
In region Si. The maximum logarithmic probabllily ii is then calculated as follows where n is the number of data points 

30 in the region Si. 

[0164] If the maximum logarithmic probability Ii is 1-a < li < 1+a (where a is a positive integer) .in.step S3, the area 
Is detennined to be a lane In step S4. For example, region S1 is detennlned to be a lane because the distribution 
pattems are similar although the brightness levels of pattems a and b in Fig. 32 differ. 
[0165] If the above equation is not true, the area is determined not to be a lane at step S5. 
35 [01 66] If i > N x N at step S6, the process ends, if not, I is incremented to 1 + 1 at step S7, and the procedure loops 
back to step S2. 

[0167] As described above, the road area extraction apparatus of the present embodiment detects an area with the 
same brightness pattern as the area directly below the vehicle In the brightness histogram to accurately extract the 
lane area. The overall image can also be segmented based on a probability comparison with the SO of the brightness 
40 pattem. 

[01 68] A second alternative of a road area extraction apparatus according to the second embodiment of the present 
invention is described next below with reference to Figs. 34 and 35. This road area extraction apparatus comprises a 
digital image pickup 31 , brightness histogram generator 32, and lane image extractor 33. 
[0169] The digital image pickup 31 is typically a video camera used for capturing an image of the road in front of the 

45 vehicle as described in the preceding atematives. 

[0170] The brightness histogram generator 32 produces a brightness histogram from the Input road image data. 
Using this brightness histogram, the brightness histogram generator 32 also detects the vehicle position In the Image, 
i.e., detects the luminance at the bottom middle part of the image, In this histogram, the number of pixels having a 
brightness level greater than the brightness at the vehicle position and corresponding to the road area is substantially 

50 constant. This pixel count Is defined as pixel count g below. Note that this pixel count g also varies according to such 
factors as the camera height and view angle. 

[01 71 ] The lane image extractor 33 detennines the brightness range containing pixels corresponding to the road. 
The pixels of the road image In the brightness range determined to be the lane area of the brightness histogram are 
extracted to obtain the lane area. 
55 [01 72] The operation of the above road area extraction apparatus is described below with reference to the flow chart: 

in Fig. 36. 

[0173] The first step is to generate a brightness histogram from the luminance data captured by the digital Imaging 
pickup (step SI). The brightness histogram shown in Fig. 35 is exemplary of this histogram. 
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[0174] Using the brightness bO at the bottom center of the Image, the sum s of the frequency, or number of pixels 

observed is obtained (step S2). . i. • ^ , , oo\ 

[01751 Usingthefrequency a of pixels with a higher brightness level, the sum s of the frequency is obtained (step S3). 
[01761 The lane area is detennined to have been extracted if s is greater than the previously obtained pixel count g 
corresponding to the lane area. If s is less than g, the procedure loops back to step S5 and S3, and the frequency ai 
of brightness bO + Db Is obtained. If s Is greater than g, the procedure advances to step S5 and S6. The brightness 
b1 and original brightness bO at the vehicle position are then used to extract the pixels of brightness br where bO < br 
Sb1. 

[0177] These pixels are then located on the road image to obtain the lane area (step S7). 
[01781 The brightness values of the road use the common tendency of a road to have a continuous brightness level 
that is lower than the brightness Immediately before the vehicle. The brightness immediately below the vehicle is 
therefore obtained from a brightness histogram, the brightness values containing the frequency of the number of pixels 
in the lane area of the image, which is previously determined, is obtained from this brightness value before the vehicle, 
and the lane area can thus be accurately extracted. 

[0179] A third alternative of a road area extraction apparatus according to the second embodiment of the present 
Invention Is described next below with reference to Figs. 37 to 39. This road area extraction apparatus comprises a 
digital image pickup 41 , brightness histogram generator 42, image separator 43, lane area clustering unit 44, and lane 

width detector 45. j. ^ ,»u 

[01 80] The digital image pickup 41 is typically a video camera used for capturing an image of the road in front of the 
vehicle as described in the preceding alternatives. 

[0181] The brightness histogram generator 42 produces a brightness histogram from the input road image data as 
shown in Fig. 38. Using this brightness histogram, the brightness histogram generator 42 also detects the vehicle 
position in the Image, i.e., detects the luminance at the bottom middle part of the image. 

[01 82] The Image separator 43 then detects all valleys in from the brightness distribution, producing a segmented 
lane Image as shown in Fig. 39. , . < 

[0183] Based on the lane width information from the lane width detector 45, the lane area clustering unit 44 merges 
the segment below the vehicle with the first horizontal segment, and then with a segment in the vertical direction, The 
lane width detector 45 obtains the lane width from the image height based on the lane area obtained from the lane 
area clustering unit 44. 

[01 84] The road area extraction apparatus thus comprised operates as described below with reference to the flow 

chart in Figs. 40 and 41 . , . j u 

[0185] The first step is to generate a brightness histogram as shown in Fig. 38 from the luminance data captured by 
the digital imaging pickup (step SI). 

[01 86] Using the brightness bO at the bottom center of the image, the position in the histogram con-espondmg to the 

road surface directly below the vehicle is obtained (step S2). A valley on the left side is detected first. 

[0187] Mean brightness values b^.^^l and b^,,„2 are then obtained for the overiapping brightness regions bO - b1 

S b < bO and bO - 3b1/2 < b < bO - b1/2 where b is a brightness parameter and b. Is a constant brightness level (step S3). 

[0188] The equation b^^^a = h^,^^ - b^,^„^ is then calculated (step S4), and the sign of b^^^a Is then detennined 

(step S5) A valley is detected by detennining when the sign changes from negative to positive. 

[0189] If the sign has not changed from negative to positive, bo = bo - bi/2 (step S6), and the procedure loops back 

to step S3 (step 86). This process Is repeated to the left side of the histogram (step 87), and the procedure then 

advances to step 58. 

[0190] A similar process is then executed to detect a valley on the right. 

[0191] The first step (step SB) is to reinitialize bo to the value detected in S2, and then calculate b^^^3 and b^^an* 
forthe regions bo < b <; bo + b^ and bo + bi/2 < b ^ bo + 3bi/2. The equation b^= b b^ean* - bmeanS 's then obtained 

[0^92^^ A valley Is detected by detennining when the sign changes from negative to positive (step S10, Fig. 41). If 
the sign has not changed from negative to positive, bo = bo + b,/2 (step S11) , and the procedure loops back to step 
S7 (step S11). This process is repeated until a valley is detected to the right side. 

[0193] Whether valleys have been detected to the right side of the histogram is then detennined (step SI 2). If not, 
steps S11 and 88 are repeated. If step 812 returns YES, the pixels in the brightness region between the detected 
valleys are then detected as the lane area candidate region of the image (step S1 3). Fig. 39 represents the input image 

separated Into the areas between the detected valleys. 

[01941 The area directly below the vehicle is then detected (step S14). Region. SO In Fig. 39 is this target region. 
Target region SO is then combined with the region S^ adjacent thereto (step S15). 

[0195] The previously obtained lane width wf(h) is then referenced to calculate horizontal width w(h) (step S1 6). 
[01 96] If wf (h) > w(h) (step 51 7), the procedure loops back to step S1 5 If wf S w (step S1 7) and the height h of the 
combined regions is greater than a defined height hO, the procedure temiinates. 
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[0197] Note that hO is the height of the vanishing point of the road, which is a constant height determined by the 

instaliation height of the digital image picl<up 41 if the road Is level. If h < hO, the combined region is combined with the 
region adjacent thereto above and narrower than the top width of the lower combined region; in Fig. 39 regions S2 and 
S3 are combined. The procedure finally terminates when the height h = hO. 
5 [0198] By thus segmenting the image based on the lane width, and using i<nown properties of the road configuration 
in the Image to extract the lane area, accurate lane area extraction is possible. Note that the lane area can also be 
extracted by segmenting the image using a brightness pattern comparison In combination with the method described 
above. 

[0199] A fourth alternative of a road area extraction apparatus according to the second embodiment of the present 
10 invention is described next below with reference to Figs. 42 to 44. This road area extraction apparatus comprises a 

digital Image picicup 51 , brightness histogram generator 52, image separator 53. lane area clustering unit 54, lane 

width detector 55, lane region correlation detector 56, and lane extending direction detector 57. 

[0200] The digital image pickup 51 Is typically a video camera used for capturing an Image of the road in front of the 

vehicle as described in the preceding alternatives. 
IS [0201] The brightness histogram generator 52 produces a brightness histogram from the Input road image data as 

shown in Fig. 38. Using this brightness histogram, the brightness histogram generator 52 also detects the vehicle 

position In the image, I.e., detects the luminance at the bottom middle part of the image. 

[0202] The Image separator 53 then detects all valleys In from the brightness distribution. 

[0203] Based on the lane width Infonnatlon from the lane width detector 55, the lane area clustering unit 54 merges 
20 the segment below the vehicle with the first horizontal segment, and then with a segment in the vertical direction. The 

lane width detector 55 obtains the lane width from the Image height based on the lane area obtained from the lane 

area clustering unit 54. 

[0204] The lane region correlation detector 56 calculates the correlation between the lane area detected by the lane 
area clustering unit 54, and a triangle defined by a particular point on the vanishing line and two lane edge positions 
25 at the bottom of the Image supplied from the lane width detector 55. 

[0205] The lane extending direction detector 57 detects the direction of the lane relative to the vehicle from the point 
on the vanishing line providing the highest correlation value. 

[0206] The road area extraction apparatus thus comprised operates as described below with reference to the flow 
chart In Fig. 42. 

30 [0207] The lane width Is first detected (step S1 using the method of the third alternative of the second embodiment 

described above, 

[0208] A point r on the vanishing line is then initialized too, i.e., to the coordinate value atthe left edge of the vanishing 
line (step S2). 

[0209] The con-elation between the extracted lane area and the triangle fonned using point r - 0, i.e., the number of 

3S matching pixels, is then calculated. 

[021 0] Point r Is then Incremented to r = r +1 (step S4), thus moving the vanishing point one pixel to the right. If r 
does not exceed pixel rti at the right edge of the image (step SB), the procedure loops bacic to step S3 to repeat the 
correlation calculation. If r is beyond the right edge, the value of r resulting in the highest correlation is detected (step 
86). If r Is greater than half (rh/2) the number of horizontal pixels in the Image, the lane Is to the right of the vehicle, if 

40 r < rh/2, the lane is to the left of the vehicle. 

[021 1 ] By thus obtaining the correlation between the detected lane area and a triangle fonned between the vanishing 
point and the lane edges, the direction of the lane relative to the vehicle can be easily obtained using the maximum 
correlation value. 

[0212] A local positioning apparatus according to the first embodiment of the Invention can thus separately detect 
45 the contour points of the road area and lane marker candidates using low and high frequency Images with no spatial 
frequency overiap. The effects of noise can therefore be reduced when compared with lane extraction using only a 
high frequency Image, the method of the prior art:, and lane mariners can be detected with high precision. 
[021 3] The lane area can also be accurately detected in the road Image by means of the road area extraction appa- 
ratus according to the second embodiment of the invention by detecting the lane area from the image data, and detecting 
so a lane area polygon starting from the bottom center of the image, a point con-esponding to directly In front of the vehicle. 
[0214] The lane area can also be accurately detected in the road image by detecting the area containing the lumi- 
nance values of the pixels directly in front of the vehicle from a brightness histogram, and using the image area cor- 
responding thereto to detect the edges of the lane area starting from the bottom center of the Image, a point corre- 
sponding to directly in front of the vehicle, 
ss [0215] The lane area can also be accurately detected by detecting patterns identical to the brightness distribution 
pattem directly before the vehicle. 

[0216] The lane area can also be accurately detected by detecting the area containing the luminance values of the 
pixels directly in front of the vehicle from a brightness histogram, calculating the brightness values containing the 
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frequency of the number of pixels in the iane area of the image, which is previously detenmined, and defining the pixels 

in this area as the lane area. 

[02171 The lane area can also be accurately detected by calculating image, segments by a road width and luminance 
pattern comparison, and using known features of the road configuration in the image to extract the lane area. 
5 [0218] The invention being thus described, it will be obvious that the same may be varied In many ways. Such 
variations are not to be regarded as a departure from the spirit and scope of the Invention, and all such modifications 
as would be obvious to one sidlled In the art are Intended to be Included within the scope of the following claims. 



10 Claims 

1. A road area detection method for extracting a road area as a polygonal area In an image plane centered on a 

vehicle, comprising: 

IS a road area extraction method for segmenting a road area using road image data; and 

a road polygon detection method for a polygonal area including a point directly before a vehicle In an image 
plane from said areas separated by said road area extraction method. 

2. A road area extraction apparatus comprising: 

20 

a road imaging device for capturing an Image of a road before a vehicle; 

a brightness histogram generating means for creating a brightness histogram using said road image data from 
said road Imaging device; 

a road brightness separating means for separating from a brightness histogram said area containing luminance 
2s values immediately before said vehicle and contained between valleys in said brightness histogram generated 

by said brightness histogram generating means; and 

a road polygon detecting means for detecting a polygonal area Including a point directly before a vehicle In 
an image plane in said areas separated by said road brightness separating means; 

30 wherein said road area Is extracted as a polygonal area In an image plane centered on a vehicle. 

3. A road area extraction apparatus comprising: 

a road imaging device for capturing an Image of a road before a vehicle; 
35 a brightness pattern detection means for detecting said surface brightness pattern immediately before said 

vehicle using said road image data from said road imaging device; and 

a brightness pattern separation means for separating road areas with a brightness pattern slmllarto said road 
brightness pattem immediately before said vehicle supplied from said brightness pattern detection means, 
wherein a road area is separated from said Image by measuring said brightness pattern of said road surface. 

40 

4. A road area extraction apparatus comprising: 

a road Imaging device for capturing an image of a road before a vehicle; 

a brightness histogram generating means for creating a brightness histogram using said road image data from 

4s said road Imaging device; and 

a pixel count road area extraction means for setting on said brightness histogram a range of a predetemilned 
pixel count constituting a road area using a brightness position directly before a vehicle using said brightness 
histogram generated by said brightness histogram generating means, wherein a brightness area is set in said 
brightness histogram using a brightness value immediately before said vehicle by first obtaining said pixel 

so count of said road area to separate road areas In said Image. 

5. A road area extraction apparatus comprising: 

a road imaging device for capturing an image of a road before a vehicle; 
55 an image separating means for segmenting an Image from road Image data from said road imaging device 

based on Image characteristics; 

a road area combining means for combining a segmented area of said vehicle position In said image with 
areas adjacent thereto on both sides for said road width only using road width data from a road width detection 
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means described below and segment data from said image separating means, and repeating this operation 
to a partlcuiar position toward said top of said image to extract a road area; and 

a road width detection means forcalculating said road width at various heights in said image from said extracted 
road areas from said road area combining means, wherein image segments are combined to extract said road 
area based on road width information detected from said previous image. 

A road area extraction apparatus according to claim 5, wherein said road area combining means combines seg- 
mented areas at said vehicle position in said lateral direction to said road width, and combines areas vertically 
such that said upper area has a width less than said width of said lower area. 

A road direction detection apparatus comprising: 

a road imaging device for capturing an image of a road before a vehicle; 

an image separating means for segmenting an image from road Image data from said road imaging device 
based on image characteristics; 

a road area combining means for combining a segmented area of said vehicle position In said image with 
areas adjacent thereto on both sides for said road width only using road width data from a road width detection 
means described below and segment data from said image separating means, and repeating this operation 

to a particular position toward said top of said image to extract a road area; 

a road width detection means forcalculating said road width at various heights in said image from said extracted 
road areas from said road area combining means; 

a road area correlation detection means for calculating a correlation between a triangle defined by a vanishing 
point at a selected position on a particular vanishing line and two road edge data points at said Image bottom 

determined by said road width detection means, and an extracted road area supplied by said road area com- 
bining means, and detecting said vanishing point yielding said greatest correlation value; and 
a road direction detection means for detecting said direction of said road to said front of said vehicle from said 
vanishing point position yielding said greatest correlation value as detected by said road area correlation 
detection means, wherein the direction of said road relative to the vehicle is detected by obtaining said corre- 
lation between an extracted road area and a triangle defined by a vanishing point at a selected position on a 
particular vanishing line and two road edge data points. 
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